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Increasing  Scale  In  Military  Systems 


Increasingly  Complex  Systems 

•  ultra-large,  network-centric,  real-time, 
cyber-physical-social  systems 

—  thousands  of  platforms,  sensors,  decision 
nodes,  weapons,  and  warfighters 

—  connected  through  heterogeneous  wired 
and  wireless  networks 


Goal:  Information  Dominance 

•  Transient  and  enduring  resource 
constraints  and  failures 

•  Continuous  adaptation 

—  changes  in  mission  requirements 

—  changes  in  operating  environments 

—  changes  in  force  structure 

—  perpetual  systems’  evolution 

—  addition  of  new  systems 

•  Sustainable  -  legally,  technically,  politically 
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A  Reason  for  Concern 


Such  systems  are  going  to  be  larger  and 
more  complex  than  any  previously  seen 

•  very  serious  technical  challenges, 
obvious  and  undoubtedly  to-be- 
discovered 

•  many  vendors,  many  technologies, 
many  systems 

•  evolving  doctrine  +  evolving  technology 
+  (or  =>?)  ill-defined  requirements 


The  US  Army  is  concerned  that  the  scale 
of  future  systems  is  beyond  our  reach. 
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The  Challenge 


The  Honorable 
Claude  M.  Bolton,  Jr. 


Software  Engineering  Institute 


“Our  soldiers  depend  on  software  and  will 
depend  more  on  software  in  the  future. 

The  Army’s  success  depends  on  software 
and  the  software  industry. 

We  need  better  tools  to  meet  future 
challenges,  and  neither  industry  nor 
government  is  working  on  how  to  do  things 
light-years  faster  and  cheaper. 

How  can  future  systems,  which  are  likely  to 
be  a  billion  lines  of  code,  be  built  reliably 
if  we  can’t  even  get  today’s  systems  right?” 

—  Asst  Sec  Army  Claude  Bolton 
August  16,  2005 
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Ultra-Large-Scale  (ULS)  Systems  Study 


Gather  leading  experts  to  study: 

•  characteristics  of  ULS  systems 

•  challenges  and  breakthroughs  required 

•  promising  research  and  approaches 

Intended  outcomes: 

•  ULS  System  Research  Agenda 

•  program  proposal 

•  collaborative  research  network 


About  the  Effort 

Funded  by  the  Army  (ASA  ALT) 

Staffing:  9  member  SEI  team 

13  member  expert  panel 

Duration:  one  year  (04/05  -  05/06) 
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SEI  Team 


Linda  Northrop  Rick  Kazman 

Software  Design  and  Architecture,  Adaptive  Architectures,  Design  Methods 

Software  Product  Lines 


Peter  Feiler  Mark  Klein 

Methodologies,  Real-time  Performance  Analysis, 

Configuration  Management  Software  Architecture  Design  and  Analysis 


John  Goodenough 

Software  Reliability, 
Safety  Assurance 


Mark  Pleszkoch 

Rigorous  Software  Engineering  Methods 


Rick  Linger  Kurt  Wallnau 

Rigorous  Software  and  Software  Components, 

Systems  Engineering  Program  Generation,  Language  Semantics 


Tom  Longstaff 

Security  and  Survivability  Engineering 
in  Complex  Systems 


Bill  Pollack 

Chief  Editor 


Daniel  Pipitone 

Chief  Graphical  Designer 


— ICGSE  2006:  Scale  Changes  Everything 

Software  Engineering  Institute  CarnegieMellon  Linda  Northrop,  Oct  2006 

©  2006  Carnegie  Mellon  University 


SEI  Team  (and  Expert  Panel) 


Gregory  Abowd 

Georgia  Institute  of  Technology 


Peter  Neumann 

SRI  International  Computer  Science  Laboratory 


Carliss  Baldwin 

Harvard  Business  School 


Douglas  Schmidt 

Vanderbilt  University 

Mary  Shaw 

Carnegie  Mellon  University 


Bob  Balzer 

Teknowledge  Corporation 


Gregor  Kiczales 

University  of  British  Columbia 


Dan  Siewiorek 

Richard  P.  Gabriel  Carnegie  Mellon  University 

Sun  Microsystems 

Kevin  Sullivan 

University  of  Virginia 


AM  Mili 

New  Jersey  Institute  of  Technology 


John  Lehoczky  jack  Whalen 

Carnegie  Mellon  University  PARC 
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How  This  Study  Is  Different? 


It  presents  an  overall  research  agenda  --  not  just  for  new  tools  or  a 
new  software  method. 

It  is  based  on  the  challenges  associated  with  ultra-large  scale. 

It  focuses  on  the  future. 

It  involves  an  multi-disciplinary  base. 

It  takes  a  fresh  perspective  on  the  development,  deployment, 
operation,  and  evolution  of  software-intensive  systems. 


\ 

Germs  of  these  ideas  are  present  today  in  small  research  pockets; 
these  efforts  are  currently  too  small  to  have  much  impact  on  next- 
generation  DoD  ULS  systems. 
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ULS  Systems  Research  Agenda 


Describes 

•  the  characteristics  of  ULS 
systems 

•  the  associated  challenges 

•  promising  research  areas 
and  topics 

Is  based  on  a  new  perspective 
needed  to  address  the 
problems  associated  with 
ultra-large-scale  systems. 
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Working  Inside  and  Outside  the  Box 


Classical  Reductionism 


Define 

Characteristics 


Propose 

Research 


Identify 

Challenges 
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Ultra-Large  Scale 


Ultra-large  size  in  terms  of 

•  Lines  of  code 

•  Amount  of  data  stored,  accessed, 
manipulated,  and  refined 

•  Number  of  connections  and 
interdependencies 

•  Number  of  hardware  elements 

•  Number  of  computational  elements 

•  Number  of  system  purposes  and  user 
perception  of  these  purposes 

•  Number  of  routine  processes, 
interactions,  and  “emergent  behaviors” 

•  Number  of  (overlapping)  policy  domains 
and  enforceable  mechanisms 

•  Number  of  people  involved  in  some  way 


— ICGSE  2006:  Scale  Changes  Everything 

Software  Engineering  Institute  CarnegieMellon  Linda  Northrop,  Oct  2006 

©  2006  Carnegie  Mellon  University 


What  Is  an  Ultra-Large-Scale  (ULS)  System? 


A  ULS  System  has  unprecedented  scale  in  some  of  these  dimensions: 

•  Lines  of  code 

•  Amount  of  data  stored,  accessed,  manipulated,  and  refined 

•  Number  of  connections  and  interdependencies 

•  Number  of  hardware  elements 

•  Number  of  computational  elements 

•  Number  of  system  purposes  and  user  perception  of  these  purposes 

•  Number  of  routine  processes,  interactions,  and  “emergent  behaviors” 

•  Number  of  (overlapping)  policy  domains  and  enforceable  mechanisms 

•  Number  of  people  involved  in  some  way 

ULS  systems  will  be  interdependent  webs  of  software-intensive  systems, 
people,  policies,  cultures,  and  economics. 

ULS  systems  are  systems  of  systems  at  internet  scale. 
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Scale  Changes  Everything 


Characteristics  of  ULS  systems  arise  because  of  their  scale. 

•  Decentralization 

•  Inherently  conflicting,  unknowable,  and  diverse  requirements 

•  Continuous  evolution  and  deployment 

•  Heterogeneous,  inconsistent,  and  changing  elements 

•  Erosion  of  the  people/system  boundary 

•  Normal  failures 

•  New  paradigms  for  acquisition  and  policy 


d  k 

These  characteristics  may  appear  in  today’s  systems  and  systems  of  systems, 

but  in  ULS  systems  they  dominate. 

These  characteristics  undermine  the  assumptions  that  underlie  today’s 
software  engineering  approaches. 
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Today’s  Approaches 


The  Engineering  Perspective  -  for  large  scale  software-intensive  systems 

•  largely  top-down  and  plan-driven 

•  requirements/design/build  cycle  with  standard  well-defined  processes 

•  centrally  controlled  implementation  and  deployment 

•  inherent  validation  and  verification 

The  Agile  Perspective  -  proven  for  smaller  software  projects 

•  fast  cycle/frequent  delivery/test  driven 

•  simple  designs  embracing  future  change  and  refactoring 

•  small  teams  and  retrospective  to  enable  team  learning 

•  tacit  knowledge 


r  \ 

Today’s  approaches  are  based  on  perspectives  that  fundamentally  do  not  cope 
with  the  new  characteristics  arising  from  ultra-large  scale. 
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A  New  Perspective  is  Required 


“The  older  is  not  always  a  reliable  model  for  the  newer,  the 
smaller  for  the  larger,  or  the  simpler  for  the  more 
complex... Making  something  greater  than  any  existing  thing 

necessarily  involves  going  beyond  experience.” 

Henry  Petroski 

Pushing  the  Limits:  New  Adventures  in  Engineering 


INSIGHT 


— ICGSE  2006:  Scale  Changes  Everything 

Software  Engineering  Institute  CarnegieMellon  Linda  Northrop,  Oct  2006 

©  2006  Carnegie  Mellon  University 


Today  We  Build  “Buildings” 
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We  Need  To  Think  Cities 
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We  Need  to  Think  Ecosystem 


Ivon  p  l„,VN  D  EtOS YSjTBM 


Diverse  users  with  complex  networked 
dependencies  and  intrinsic  adaptive 
behavior 

Has: 

•  Robustness  mechanisms:  achieving 
stability  in  the  presence  of  disruption 

•  Measures  of  health:  diversity, 
population  trends,  other  key  indicators 
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We  Need  to  Think  Socio-Technical 
Ecosystems 


Socio-technical  ecosystems  include 
people,  organizations,  and  technologies 
at  all  levels  with  significant  and  often 
competing  interdependencies. 


There  will  be  organizations  and 
participants  responsible  for  producing 
ULS  systems. 


There  will  be  competition  for 
resources. 


There  will  be  organizations  and 
participants  responsible  for  setting 
policies. 


•  There  will  need  to  be  local  and  global 
indicators  of  health  that  will  trigger 
necessary  changes  in  policies  and  in 
element  and  system  behavior. 
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Why  a  New  Perspective? 


There  are  fundamental  assumptions  that  underlie  today’s  software 
engineering  and  software  development  approaches  that  are 
undermined  by  the  characteristics  of  ULS  systems. 

There  are  challenges  associated  with  ULS  systems  that  today’s 
perspectives  are  very  unlikely  to  be  able  to  address. 
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ULS  Systems  vs  Today’s  Approaches  - 1 


Decentralized  control 

All  conflicts  must  be  resolved  and 
resolved  centrally  and  uniformly. 

Inherently  conflicting,  unknowable,  and  diverse 
requirements 

Requirements  can  be  known  in 
advance  and  change  slowly. 

Tradeoff  decisions  will  be  stable. 

Continuous  evolution  and  deployment 

System  improvements  are  introduced 
at  discrete  intervals. 

Heterogeneous,  inconsistent,  and 
changing  elements 

Effect  of  a  change  can  be  predicted 
sufficiently  well. 

Configuration  information  is  accurate  and 
can  be  tightly  controlled. 

Components  and  users  are  fairly 
homogeneous. 

ULS  Systems  vs  Today’s  Approaches  -  2 


Characteristics 

Today’s  assumptions  undermined 

Erosion  of  the  people/system  boundary 

People  are  just  users  of  the  system. 

Collective  behavior  of  people  is  not  of  interest. 

Social  interactions  are  not  relevant. 

Normal  failures 

Failures  will  occur  infrequently. 

Defects  can  be  removed. 

New  paradigms  for  acquisition  and  policy 

A  prime  contractor  is  responsible  for  system 
development,  operation,  and  evolution. 
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Challenges 


ULS  systems  will  present  challenges 
in  three  broad  areas: 

•  Design  and  evolution 

•  Orchestration  and  control 

•  Monitoring  and  assessment 
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Design  and  Evolution 


Specific  challenges  in  ULS  system  design  and  evolution  stemming  directly 
from  the  characteristics  of  ULS  systems: 

•  Economics  and  industry  structure 

•  Social  activity  for  constructing  computational  environments 

•  Legal  issues 

•  Enforcement  mechanisms  and  processes 

•  Definition  of  common  services  supporting  the  ULS  system 

•  Rules  and  regulations 

•  Agility 

•  Handling  of  change 

•  Integration 

•  User-controlled  evolution 

•  Computer-supported  evolution 

•  Adaptable  structure 

•  Emergent  quality 
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Orchestration  and  Control 


Orchestration  and  control  refers  to  the  set  of  activities  needed  to  make 
the  elements  of  a  ULS  system  work  together  in  reasonable  harmony  to 
ensure  continuous  satisfaction  of  mission  objectives. 

Orchestration  is  needed  at  all  levels  of  ULS  systems  and  challenges  us 
to  create  new  ways  for 

•  Online  modification 

•  Maintenance  of  quality  of  service 

•  Creation  and  execution  of  policies  and  rules 

•  Adaptation  to  users  and  contexts 

•  Enabling  of  user-controlled  orchestration 
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Monitoring  and  Assessment 


The  effectiveness  of  ULS  system  design,  evolution,  orchestration,  and 
control  has  to  be  evaluated. 

There  must  be  an  ability  to  monitor  and  assess  ULS  system  state, 
behavior,  and  overall  health  and  well  being. 

Challenges  include 

•  Defining  indicators 

•  Understanding  why  indicators  change 

•  Prioritizing  the  indicators 

•  Handling  change  and  imperfect  information 

•  Gauging  the  human  elements 
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Where  Do  We  Focus  Our  Research 


•  Address  the  predominant 
characteristics  of  ULS  systems  and 
the  three  challenge  categories. 

•  Look  for  breakthroughs  not 
incremental  improvement  in  current 
approaches. 

•  Take  a  more  expansive  view  of 
software  research  and  include  its 
interactions  with  associated  research 
in  the  physical  and  social  sciences. 
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Potential  Tools:  Game  Theory 


Research  Areas 


GAME  THEORY 

Oskar  Morgenstern 

1902-1977 


Monograph  Efficiency  y,3 

Marginal  Rate  of  Reference  0.0 

Public  Perception  Indicator  1.8 

Productivity  Pom  aria  I  In  flax  -24.0 

Expected  Utility  1.6 


John  von  Neumann 

1903-1957 

Monograph  Efficiency  3.7 

Marginal  Rate  of  Reference  12.0 

Public  Per  Coptic  a  Indicator  22.3 

Productivity  Par  a  nr  in  I  Index  -44.0 

Expected  Utility  1.0 


John  F.  Nash  Jr. 

L1928 

Monograph  Efficiency  0.0 

Marginal  Rate  of  Reference  0.0 

Public  Perception  Indicator  11.1 

Productivity  Potential  Index  +10.2 

Expected  Utility'  5-0 


Reinhard  Selten 

b193G 


Monograph  Efficiency 

6.3 

Marginal  Rate  of  Reference 

14.0 

Public  Perception  Indicator 

2.2 

Productivity  Pore  nr  in  1  Index 

+  11.4 

Expected  Utility 

2.0 

Algorithmic  Mechanism  Design 

•  games  +  microeconomics  +  computation 

•  computational  markets  for  any  scarce 
ULS  resource? 


Institution  Design 

•  learning  games  +  self-reinforced 
expectations  +  cultural  norms 

•  better  formal  models  of  acquisition  in 
non-prime-dominated  landscape? 
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Networks,  Statistical  Mechanics, 
Complexity 


Networks  Are  Everywhere 

Recurring  “scale  free”  structure 
•  internet  &  yeast  protein  structures 


Unstable  Equilibrium 

How  many  changes  before  a 
system  becomes  unstable? 


Analogous  dynamics?  What  scale  and  frequency  of 

•  epidemiology,  robustness  and  disruptions  can  be  expected? 

vulnerability 
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Economics  (Finance)  As  Design  Criteria 


nr 


XXX 

X  X 
X  X 


Design 

Rules 


Design  Rules  Task  Group 


Drive 

System 


Main 

Board 


LCD 

Screen 


Pack¬ 

aging 


.XX  X 

X  .  X  X  JL 

X  X  .  x^^™ 

X  X  X  .  X  X 

X  X  .  X 

X  X  X  X  . 


JL 

X 

X 

X 

X 

X 

I— 

X 

X 

X 

X  X 
X 
X 

X 

X  X 


X  X 

X  X 


X  X 

X  .  X  X 
X  X  .  X  X  X  X 

X - X^,  X  X 

X  X  X  X  .  X  X 

X  X  X  X  .  X 

X  X  X  X  .  X 
XX  X  . 


Hidden  Modules 
many  Task  groups 


xx  x 
.xxx 
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x  x  ►.  x 
xxx. 
x  xx 
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.  x 


x  .  x 
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X  X 
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X 

.  X 
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X 

.  X 


X 

X 

X 
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Design  rules  (feature 
parameterization) 

Module  maximizes  option 
value 

ROI  inversely  proportional 
to  module  footprint 

Each  module  a  potential 
point  for  competition 

(apologies  for  abuse  of 
ideas  to  Carliss  Baldwin) 
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System 
Testing 
&  Integ¬ 
ration 


x 
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X 
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System 
x  Integration 
and  Testing 
x  Task  Group 
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Research  Portfolio 


We  recommend  an  interdisciplinary 
portfolio  of  seven  research  areas 
and  suggested  topics  for 
breakthrough  research  needed  to 
meet  the  challenges  associated  with 
ULS  systems. 

•  Is  not  expressed  in  terms  of 
today’s  “hot”  technologies. 

•  Does  not  supplant  current 
software  research. 

•  Expands  today’s  horizons. 
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Research  Areas  - 1 


Researc  h  Areas 


6.1  Human  Interaction:  Involves 

anthropologists,  sociologists,  and  social 
scientists  conducting  detailed  socio- 
technical  analyses  of  user  interactions  in 
the  field,  with  the  goal  of  understanding 
how  to  construct  and  evolve  such  socio- 
technical  systems  effectively. 

•  Context-Aware  Assistive  computing 

•  Understanding  Users  and  Their  Contexts 

•  Modeling  Users  and  User  Communities 

•  Fostering  Non-Competitive  Social 
Collaboration 

•  Longevity 
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Research  Areas  -  2 


6.2  Computational  Emergence: 

Explores  the  use  of  methods  and 
tools  based  on  economics  and  game 
theory  (e.g.,  mechanism  design)  to 
ensure  globally  optimal  ULS  system 
behavior  by  exploiting  the  strategic 
self  interests  of  the  system’s 
constituencies;  explores 
metaheuristics  and  digital 
evolution  to  augment  the  cognitive 
limits  of  human  designers. 

•  Algorithmic  Mechanism  Design 

•  Metaheuristics  in  Software 
Engineering 

•  Digital  Evolution 
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Research  Areas  -  3 


HtWtrdi  Areas 


6.3  Design:  Broadens  the  traditional 
technology-centric  definition  of  design 
to  include  people  and  organizations; 
social,  cognitive,  and  economic 
considerations;  and  design  structures 
such  as  design  rules  and  government 
policies. 

•  Design  of  All  Levels 

•  Design  Spaces  and  Design  rules 

•  Harnessing  Economics  to  Promote 
Good  Design 


•  Design  Representation  and  Analysis 


•  Assimilation 


•  Determining  and  Managing 
Requirements 
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Research  Areas  -  4 


6.4  Computational  Engineering:  Focuses  on 
evolving  the  expressiveness  of 
representations  to  accommodate  the 
semantic  diversity  of  many  languages  and 
focuses  on  providing  automated  support  for 
computing  the  evolving  behavior  of 
components  and  their  compositions. 

•  Expressive  Representation  Languages 

•  Scaled-Up  Specification,  Verification,  and 
Certification 

•  Computational  Engineering  for  Analysis  and 
Design 
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Research  Areas  -  5 


Research  Areas 


6.5  Adaptive  System  Infrastructure:  Investigates  integrated 

development  environments  and  runtime  platforms  that  will  support 
the  decentralized,  “always-on,”  nature  of  ULS  systems  as  well  as 
technologies,  methods,  and  theories  that  will  enable  ULS  systems 
to  be  developed  in  their  deployment  environments. 

•  Decentralized  Production  Management 

•  View-Based  Evolution 

•  Evolutionary  Configuration  and  Deployment 

•  In  Situ  Control  and  Adaptation 
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Research  Areas  -  6 


6.6  Adaptable  and  Predictable  System  Quality:  Focuses  on  how  to 
maintain  quality  in  a  ULS  system  in  the  face  of  continuous  change, 
ongoing  failures,  and  attacks  and  how  to  identify,  predict,  and 
control  new  indicators  of  system  health  (akin  to  the  U.S.  gross 
domestic  product)  that  are  needed  because  of  the  scale  of  ULS 
systems. 

•  Robustness,  Adaptation,  and  Quality  Attributes 

•  Scale  and  Composition  of  Quality  Attributes 

•  Understanding  People-Centric  Quality  Attributes 

•  Enforcing  Quality  Requirements 

•  Security,  Trust,  and  Resiliency 

•  Engineering  Management  at  Ultra-Large  Scales 
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Research  Areas  -  7 


6.7  Policy,  Acquisition,  and 
Management:  Focuses  on 
transforming  acquisition  policies 
and  processes  to  accommodate 
the  rapid  and  continuous  evolution 
of  ULS  systems  by  treating 
suppliers  and  supply  chains  as 
intrinsic  and  essential  components 
of  a  ULS  system. 

•  Policy  Definition  for  ULS  Systems 

•  Fast  Acquisition  for  ULS  Systems 

•  Management  of  ULS  Systems 


Software  Engineering  Institute 


Research  Areas 


ICGSE  2006:  Scale  Changes  Everything 

Carnegie  Mellon  Linda  Northrop,  Oct  2006  44 

©  2006  Carnegie  Mellon  University 


(The  Selling  Of)  Mechanism  Design: 

Game  Theory  +  Microeconomics  +  Computation 


Relevance  to  DoD  Missions 

•  ULS  systems  must  support 
war  fighters  at  all  echelons  who 
are  engaged  in  information¬ 
intensive  activities  and  who  must 
share  critical  but  finite  information 
technology  resources. 

•  Future  combat  missions  will  require  robust 

and  decentralized  resource  allocation  mechanisms  that  are  strategy  proof, 
support  a  diversity  of  interests,  and  provide  fully  predictable  and  near-optimal 
global  outcome. 

f  "\ 

Key  Concepts 

• Game  theory  provides  mathematical  tools  to  study  the  outcomes  of  interactions  among  self-interested, 
and  possibly  deceptive,  players,  where  the  interactions  are  governed  by  a  set  of  rules. 

• Mechanism  design  is  the  inverse  of  game  theory:  it  seeks  to  discover  the  rules  of  games  that  will  result 
in  a  desired  outcome  despite  self-interested  and  deceptive  behavior. 

•Mechanism  design  is  concerned  primarily  with  microeconomics  the  economic  behavior  of  agents  in  the 
face  of  scarcity. 

v _ : _ J 
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Toward  a  Roadmap  for  a  ULS  Systems 
Research  Program 


There  are  many  possible  approaches  to  structuring  a  research 
program  from  the  ULS  Systems  Research  Agenda 

We  provide  three  possible  support  structures  based  on 

1.  Specific  DoD  missions  and  capabilities 

2.  DoD  research  funding  types  required 

3.  Estimates  of  the  relative  starting  points  of  the  research 

We  expect  that  sponsors  with  different  needs  will  likely  choose  to 
support  different  combinations  of  research  and  perhaps  different  paths 
through  (or  projects  within)  the  research  program. 

f  '  '  \ 

The  envisioned  outcome  of  the  proposed  research  is  a  spectrum  of  technologies  and 

methods  for  developing  ULS  systems,  with  national-security,  economic,  and  societal 
benefits  that  far  extend  beyond  ULS  systems  themselves. 

\ _ y 
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Study  Conclusions 


There  are  fundamental  gaps  in  our  current  understanding  of  software 

development  at  the  scale  of  ULS  systems. 

These  gaps 

•  present  profound  impediments  to  the  technically  and  economically 
effective  achievement  of  the  DoD  goal*  of  deterrence  and  dominance 
based  on  information  superiority 

•  require  a  broad,  fresh  perspective  and  multi-disciplinary,  breakthrough 
research 

We  recommend 

•  a  ULS  Systems  Research  Agenda  that  includes  research  areas  based 
on  a  fresh  perspective  aimed  at  challenges  arising  from  increasing  scale 

•  short-term  startup  program  and  a  long-term,  substantive  research 
program  for  ULS  systems 

*  As  stated  in  the  Quadrennial  Defense  Review  (QDR)  Report,  Feb  2006 
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ULS  Systems  Research  Study  Report 
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The  Start  of  a  Collaborative  Research  Network 


As  a  result  of  this  study,  a  community  of  interest  in  ULS  systems  and 
the  needed  research  is  already  beginning  to  grow. 

•  Contributors  to  this  study  have  begun  to  describe  and  advocate  ULS 
system  challenges  and  potential  research. 

•  Over  the  course  of  the  past  year,  keynote  presentations  and  invited 
talks  related  to  this  study  have  been  given  at  a  diverse  set  of  forums. 

•  Workshops  and  panel  discussions  are  being  organized  and  the  term 
“ultra-large-scale  systems”  is  gaining  traction  with  others  outside  the 
original  group  charged  to  conduct  the  study. 

•  Several  organizations  and  many  individuals  have  expressed  interest  in 
getting  involved  in  a  ULS  Systems  Research  Program. 
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What  We  Learned 


•  New,  multi-disciplinary  perspective  and  new  research  in  building 
ultra-large-scale  systems  is  long  overdue. 

•  Manifestations  of  scale  and  its  attendant  complexity  arise  in  many 
disciplines,  and  can  be  understood  as  a  phenomenon  in  its  own 
right. 

•  The  ULS  Systems  research  proposal,  if  funded,  will  provide  a 
clearing  in  which  new  ideas  can  be  explored. 
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What’s  Next 


•  ULS  System  Senior  Leader  Forum 

Initiation  of  pockets  of  ULS  System 
Research 

Promulgation  of  ULS  System 
Ideas 

•  Lectures 

•  Talks 

•  Workshops 

•  Publications 

Lobbying  for  a  ULS  System 
Research  Program 
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Contact  Information 


Linda  Northrop 
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Product  Line  Systems  Program 
Telephone:  412-268-7638 
Email:  lmn@sei.cmu.edu 


U.S.  Mail: 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213-3890 

SEI  Fax:  412-268-5758 
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